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: Abstract 

O 

■ We discuss the properties of equilibrium states in an autoassociative memory 
, model storing hierarchically correlated patterns (hereafter, hierarchical pat- 

^ [ terns). We will show that symmetric mixed states (hereafter, mixed states) 

^ I are bi-stable on the associative memory model storing the hierarchical pat- 

■ terns in a region of the ferromagnetic phase. This means that the first-order 
O ■ transition occurs in this ferromagnetic phase. We treat these contents with a 

statistical mechanical method (SCSNA) and by computer simulation. Finally, 
we discuss a physiological implication of this model. Sugase et al. analyzed 
^ ' the time-course of the information carried by the firing of face-responsive neu- 

' rons in the inferior temporal cortex Q. We also discuss the relation between 

the theoretical results and the physiological experiments of Sugase et al. 

PACS numbers: 84.35. -hi, 87.10.-he, 05.90.-hm, 89.70.-hc 
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I. INTRODUCTION 



There are two kinds of hypotheses regarding internal representations of memory items 
in the brain. First, there is the "distributed representation" hypothesis, which assumes that 
our memory items are encoded by neuron activity patterns. But another hypothesis (e.g., the 
"grand mother cell" hypothesis) has been proposed, where the memory items are represented 
by the excitation of corresponding neurons: that is, the "local representation" hypothesis. 
Let us consider a neural network model consisting of N neurons. The number of memory 
items in the local representation is 0{N). One might think that the number of memory 
items in the distributed representation is 0(2^) in the case of storing binary patterns. 
However, this consideration is meaningless. If we consider error correcting abilities, this 
becomes 0{N) on the basis of the statistical mechanical theories (for example 0). One of 
the remarkable advantages of the distributed representation is that the relationship between 
the memory items is naturally implemented by the distance of memory patterns. However, 
many studies on associative memory models have been confined to those with uniformly 
distributed patterns. Thus, we should discuss a model that stores some structural patterns. 
We will extend the conventional SCSNA [|| to a generalized SCSNA in order to treat a 
model with any structural pattern and a general class of output functions of neuron. 

When an associative memory model is made to store memory patterns as a result of 
correlation learning, a pattern, which has an uniform overlap with some of the stored memory 
patterns, automatically becomes the equilibrium state of the model. This is called the mixed 
state. It is not appropriate to think that this mixed state is a side-effect and/or that it is 
unnecessary for information processing. Amari has discussed a "concept formation" using 
the stability of mixed state . The correlated attractor accounting for the physiological 
experiments by Miyashita can be interpreted as a mixed state in a broad sense. Recently, 
Parga and Rolls |^ used the mixed state in their research on the mechanism of invariant 
recognition with a coordinate transformation in the visual system. 

The aim of this research is to study the properties of mixed states in the autoassocia- 
tive memory model storing hierarchical patterns by utilizing the generalized SCSNA and 
computer simulation. First, we discuss a model that stores patterns in which a two-stage 
hierarchy exists. With the generalized SCSNA, we derive the order parameter equations 
for the equilibrium state in this model. By solving the obtained equations, we show that 
two kinds of mixed states coexist in a particular region of the retrieval phase (ferromagnetic 
phase). It is a characteristic of the two kinds of mixed states that they have different values 
of cross-talk noise variances, that is, they are influenced by the uncondensed patterns in dif- 
ferent ways. This kind of bi-stability in the retrieval phase has not been previously reported. 
We will show that the bi-stability of mixed states does not depend on the number of patterns 
in the same cluster. Generally speaking, it is not so easy to confirm the multi-stable states 
by computer simulation. However, we succeed in confirming the bi-stability of the mixed 
states by computer simulation after considering the qualitative properties of the retrieval 
dynamics in this model. Next, we treat a model storing a set of hierarchical patterns and 
uniformly distributed patterns in order to investigate universality of the bi-stability. From 
these theoretical results, we found that a bi-stability of mixed states also exists in this model. 
Contrary to these models, such bi-stability of mixed states does not exist in a model where 
the contribution of uncondensed patterns to the synaptic couplings is replaced by the spin 
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glass type interaction. We will discuss the reason for this by using the SCSNA. 

Recently, Sugase et al. have reported interesting phenomena concerning the temporal dy- 
namics of face-responsive neurons in the inferior temporal (IT) cortex [|1| . In the discussion, 
we will examine the relation between the obtained results and the physiological experiments 
by Sugase et al. 



II. MODEL 

Let us consider a recurrent neural network consisting of neurons with an output 
function F{-). We employ the synchronous dynamics, 

N 

x't'=F{Y.-h^]). (1) 

where x\ represents a state of the i-th neuron at discrete time t, and discuss the case of 
the thermodynamics limit {N — > oo). Jij in the above equation denotes a synaptic coupling 
from the j-th neuron to the i-ih neuron. In this work, we discuss the equilibrium state of 
eq. (|I|). For simplicity, we treat a two-stage hierarchy, which is one of the simplest cases. 
This can be easily extended to more complex hierarchies. One can use many procedures to 
make the set of ultrametric patterns, but we employ the following method. Each component 
of pattern ^'^'^ is a random variable drawn from the following probability distributions, 

P[ef = ±l] = ^, 2 = l,2,---,iV /i = l,2,---,p, (2) 

p[er = ±1] = ^4^, z/ = i,2,---,s (3) 

with < 6 < 1. The distance between patterns is expressed by 

E[ere'^']=5,,'(B)..,, (4) 
(B)^^, = 6^^,{6^^i + 6^(1 - 6uu')), (5) 

where E[-] stands for an average with respect to the probabilities distributed in eqs. @) and 



and 5^^/ is the Kronecker's 5 defined as. 



_ J 1 (/i = /i') 
1 ^ ^^') 



(6) 



According to the definition of the matrix B in eq. (^, B is s x s matrix. As shown in 
eq. (0), the memory patterns ^^''^ have a two-stage ultrametric structure. (B)^!,/ in eq. (H) 
stands for the element in the z/-th row, the z/'-th column of matrix B. ^^''^ are ps uniformly 
generated patterns when 6 = 0, while in the /i-th cluster are the same when 6=1. 

We employ the simple Hebbian rule as the learning rule, and the synaptic coupling Jij 
is set to 



aN 
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where a = p/N . Since the number of clusters is aN , we call a the loading rate. is not 
explicitly used for the learning rule. We discuss two kinds of models. One is the previously 
defined model with the couplings Jij given by eq. (|^), which we call Model 1. The other 
model, Model 2, is shown here with the following couplings Jjj, 

-IS -| aN 

J^^ = JrT.^"^" + J,T.m- (8) 

We will examine the stability of the memory pattern and a mixed state which has a uniform 
overlap with an odd number of memory patterns in the same cluster. There exist mixed 
states which have finite overlaps with memory patterns in different clusters. However, we 
do not discuss this topic in the present paper. 



III. A GENERALIZED SCSNA 

There are two kinds of statistical mechanical theories treating the equilibrium properties 
of associative memory models. One is based on the SCSNA proposed by Shiino and Fukai 
§. The other one is the replica theory, which has been used to analyze the equilibrium 
states in Model 1 H. However, the replica theory cannot be applied to a system where 



the free energy cannot be defined (e.g., a model with a nonmonotonic output function |T0 
or an oscillator associative memory model while the SCSNA can treat these systems 

without the energy function. Since previous studies with the SCSNA have mainly focused 
on systems with uniformly distributed patterns, the SCSNA cannot be directly applied to 
Model 1. 

We will extend the previously proposed SCSNA to a generalized SCSNA in this section 
in order to treat a model storing a set of structural patterns cr'^ (p = l,2,...,dA^) ran- 
domly generated by an independent identical probability distribution with respect to i. The 
correlation matrix (C)pp/ between the two patterns cr^ and cr'' is given by, 

1 aN 

(C)PP' = T^E«'' (9) 

i=l 

= E[afcrf']. (10) 

We consider a recurrent network consisting of neurons with the output function F{-) and 
synaptic couplings Jij given by 

1 aN 

^. = ]7E« (11) 

Since the number of memory patterns is aN , a is defined as the loading rate in this model. 
^'^''^ or B in eq. (^) is an example of cr^ or C in eq. (|10|), respectively, 

cr^'^e" p = s(/i - 1) + I/, (12) 
{C)pp> (13) 
a as. (14) 
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We consider the case where the state of neurons at discrete time step t, a;*, is synchronously 
updated, 



N 



t+1 



F{Y.J'ii^])^ (15) 



and discuss the equihbrium state x with the hmit t oo. We introduce a set of rotated 
memory patterns, a- = {iJi^, cfi^, (Ji"^}, as 

= ^ E Wpp'^f\ (16) 
V^p p'=i 

W ={w\w\---,w^^)^ . (17) 

wP{p = 1,2, ...,aN) and Kp(p = 1,2, ...,aN) in the above equations represent the p-th aN 
dimensional normalized eigenvector of matrix C and the p-th eigenvalue of matrix C for 
w'', respectively. Each component erf is statistically dependent (independent of) on /i (i), 
respectively, and satisfies the following conditions, 

^E(^n^ = i> (18) 

1=1 

1 ^ 1 

^E^M'~0(-^), p^p'. (19) 
Using the rotated patterns, we can rewrite Jij in eq. (|TT1) as 

= ]7 E (20) 



p=l 



The overlaps m'' between the equilibrium state x and cr'' are defined by the following equa- 
tion. 



m 



p 



1 ^ 



l^E^f^^- (21) 



If one assumes that the equilibrium state x has nonzero overlaps with s rotated patterns 
ct''(1 < p < s), we can derive the SCSNA order parameter equations (see Appendix A), 



mP = J Dz <aPY{z,a\a^,..,a') (22) 
q = ( Dz< {Y{z, cr\ a', .., (23) 



U = ^= / Dzz < Y{z, a\ a\ .., a') >a, (24) 
D^ = ^exp(^), (25) 



27r 
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Y{z, a\ a\ .., a') = F{J2 i^pO^m" + VY{z, a\ a^ .., a') + Vdrz), (26) 

p=i 

r = d/ (iM-^^— = — Tr - — — . (28) 



where < ■ ■ ■ >^ stands for an average over the condensed patterns ct = (a^, a^, a*). We 
can express the sum of Kp in terms of an integration along continuous eigenvalue '^(^) = 
for p, ^ oo. We remarked that the analytical expressions of r and F given by eqs. (^7]) 
and ( p8D depend only on the matrix C and do not explicitly depend on the condensed 
patterns ct''(p = l,2,...,s). This fact leads to the following order parameter equations for 

1 ^ 

the equilibrium state having nonzero overlaps = — ^ erf Xj with s original memory 

^ i=i 

patterns (t^ (p = 1, 2, s): 

mP = J Dz <aPY{z,a\a'^,..,a') >(T, (29) 
q = j Dz< {Y{z, a\ a\ .., a')f >cr, (30) 

1= [ Dzz<Y{z,a\a\..,a')>fr, (31) 



y/ar 
dz — z"^ 

Dz = -^eM^). (32) 



Y{z, a\ a^ .., a') = F{J2 o'^mF + VY{z, a\ (T^ .., a') + \fdcrz), (33) 
- '-^(t^V (34) 



aN V(I-Cf/)2;' 

^.TrfJ^V (35) 



VI -Cf/ 

Note that the off-diagonal terms of the matrix C between the condensed and uncondensed 
pattern spaces can be neglected if s is taken to be sufficiently large but 0(1). 



IV. RESULTS 

We discuss the case where F(-) = sgn(-) defined by 

^Sn(.) ^ { j: \ "I . (36) 

First, we will show the results of the analysis for the equilibrium properties in Model 1. 
Now, let us apply the generalized SCSNA proposed in section III to Model 1. We rewrited 
eq. (HD by 
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C = B0B0,...,0B. (37) 
^ ^ ^ 

p 

The matrix B has the eigenvalues Ai = 1 + (s — l)b'^,Xi, = 1 — 6^(2 < z/ < s). One 



1 



N 



assumes that the overlaps m^''^ = — ^^j^'^^Xj {u = 1, 2, s) have the values of 0(1). By 

i =l 

considering the relations given by eqs. ([121) , ([T^) and (|37|), we obtained the following SCSNA 
order parameter equations for this model: 

m''^ = < e^'-erf( ^-=^^ ) (38) 
q = 1, (39) 

where < • ■ • > stands for an average over the condensed patterns = (■C^'^,^^'^, ■ ■ ■ ,^^'^)- 
In the above equations, we omitted the F terms corresponding to eq. ( P^D by applying the 
Maxwell rule. The results coincided with those of the theoretical analysis using the replica 
theory |p. For simplicity, we will show the case where s = 3, 5. Fig. shows a phase diagram 
of numerical solutions for eqs. (|3^) - (0). The critical loading rate of a memory pattern 
and that of a mixed state are plotted against b. The triangular area shown in the figure 
represents the region where two kinds of mixed states coexist. We called this region the "bi- 
stable region". In this paper, a mixed pattern which is similar to sgn(J2t=i^^'''), is defined 
as rj'^, and the other one, which is reported here for the first time, is defined as 77^. As shown 
in 0, the SCSNA assumes the stability of equilibrium state. There is no free energy in the 
SCSNA formalism. However, we discuss the present model using the equilibrium statistical 
mechanics since it has the energy function. From the statistical-mechanical viewpoint, 
the first-order transition occurs at the dashed curve in Fig. |I]. In the region below the dashed 
curve, the free energy of rj'^ is less than that of f)^, while the free energy of rj'^ is larger than 
that of 77^ in the region above the dashed curve. 

We will examine two typical examples in the bi-stable region. We consider the case where 
the retrieval pattern is ^ ' . aj? (aj?) is defined as the critical loading rate of 77^ ( 17 ). First, 
we discuss the region in which < aj? as follows. Fig. ^(a) shows how the overlaps between 
the equilibrium state and the retrieval pattern ^ ' depend on a. Two kinds of mixed states 
coexist when 0.01500 < a < 0.01765, as shown in Fig. 0(a). From the numerical analysis, the 
value of the cross-talk noise variance expressed by r in eq. (^0]) becomes larger in the order 

T]^, 7)^ . The variation of the overlap — ^ ^I'^vl with a is larger than that of — ^ ^I'^vl 



with a, as shown in Fig. 0(a). Thus, we found that fj^ is more influenced by the uncondensed 
patterns than 77^ is. We will transform the the order parameter equations given by eqs. (|38D 
- (^1]) into an equation in one variable in order to qualitatively understand the nature of this 
bi-stability. Considering that m^'*^ is the same value for any u in the mixed states, we can 
transform the order parameter equations representing the mixed states into the equation in 
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one variable y by replacing with (see Appendix B). Fig. §(b) shows the graphical 
solutions of the transformed equation around the bi-stable region. Each intersection of the 
y-axis and function a,b = 0.61, s = 3) represents the corresponding solution. Fig. ||(b) 
shows that two kinds of mixed states, that is, r]^ and fj^, coexist at a = 0.016. On the other 
hand, there is no such bi-stability in the retrieval region except for the triangular region. 
Fig. 1 shows that bi-stability also exists for s = 5. By analyzing the graphical solutions 
of the transformed equation for various values of s, we deduced that the bi-stability of the 
mixed states exists for any value of s in Model 1. 

We performed computer simulation to confirm the bi-stability of mixed states, and ob- 
tained equilibrium states, which are expressed by the dotted curve (A) in Fig. H(a), as 
follows. We first got an equilibrium state at a = 0.0003 < with the initial state set 
to sgD.{J2t=i^^''')- After that, equilibrium states for various a were obtained by gradually 
increasing the value of a from a = 0.0003. Since the simulation results mostly agreed with 
the theoretical results, we could not distinguish between them as shown in this figure. The 
dotted curve (B) in Fig. |^(a) was obtained as follows. An equilibrium state at a = 0.019 
(a^ = 0.01765 < a < = 0.01982 ) was obtained by gradually increasing the value of a 
from a = 0.0003 in computer simulation. Using this equilibrium state, we got each equilib- 
rium state at the corresponding a by gradually decreasing the value of a from a = 0.019. 
We carried out computer simulation with = 10000; a typical example is shown in Fig. |^ 
(a). Fig. ^(a) shows a typical example (6 = 0.55, s = 3) of the case where < a^- Two 
kinds of mixed states coexisted when 0.01164 < a < 0.01389. The axes in this figure are the 
same axes in Fig. 0(a). Fig. |^(b) shows the graphical solutions of the transformed equation 
in one variable around the bi-stable region when s = 3,b = 0.55. This figure shows that two 
kinds of mixed states, that is, fj^ and rj^, are stable at a = 0.0125. The dotted curve (C) in 
Fig. H(a) was obtained as follows. We first got an equilibrium state at a = 0.0003 with the 
initial state set to sgn{J2t=i ^^''^)- After that, equilibrium states for various a were obtained 
by gradually increasing the value of a from a = 0.0003. Since r]^ is always stable when 
a'2 < a < a'2, an equilibrium state corresponding to fj^ could not be obtained by computer 
simulation with the initial state set to sgn{J2t=i ^^''^) ■ By considering the qualitative prop- 
erties of the retrieval dynamics, we carried out computer simulation in the following manner. 
Fig. ^ shows the trajectories of temporal evolutions of overlaps of the state with ^^'^ or 
^^'^ respectively, 

1 ^ 

"^'" = ^Ee^"4 (42) 

i=i 

The parameters were set to s = 3, 6 = 0.475, a = 0.0087, = 40000. The initial states were 

set as P[a;^ = ±1] = — - — with an initial overlap mg^. Theoretically, m]''^ = m]'^ 

2 

holds over the retrieval process in this case, crosses that are shown in Fig. |^ represent 
the theoretically obtained equilibrium states corresponding to fj^, rj^ and Two kinds of 
mixed states (r]^, fj^ ) and the retrieval pattern (^^'^) were stable at this parameter set ( Fig. 

The dynamics converged to fj^, r)^, ^^'^ in order as mg ^ increased, while the cross-talk 
noise variance for the equilibrium became larger in the order, ^ ' , 77^, f)^. This phenomenon 
occurred because the cross-talk noise variance of the initial stage of the retrieval process 
increased as rriQ^ decreased. The results showed that equilibrium states corresponding to fj^ 
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can be confirmed by computer simulation when the initial state has an appropriate overlap 
with the retrieval pattern. The dotted curve (D) in Fig. ^(a) was obtained as follows. We 
got an equilibrium state at a = 0.0135 where the initial state had the appropriate overlap 
with the retrieval pattern (^^'^). We obtained equilibrium states for various ot by increasing 
(decreasing) the value of a from a = 0.0135. We carried out computer simulation with 
N = 10000; Fig. 3(a) shows a typical example. Although the result from this simulation did 
not always quantitatively agree with the theory owing to spurious states, we can state that 
the bi-stability of two kinds of mixed states exists in the particular region of the retrieval 
phase from the hysteresis shown in Figs, ^(a) and|^(a). 

Let us present the equilibrium properties in Model 2. Since the uncondensed patterns 
^^(/i = 2,3, ...,aN) of Model 2 satisfy the orthogonal condition E[^f ] = 6^^', the order 
parameter equations for this model were derived by replacing eq. (|iO| ) with eq. (P5[): 



^i,u ^ < ^i.^erf( ^-=^L- ) >.i, (43) 

\'2ar s 

(44) 
(45) 

— < exp(-( V S^)') >.i . (46) 

We transformed the order parameter equations concerning the mixed states into an equation 
in one variable y'' = (see Appendix B). As in Model 1, we deduced that there is bi- 
stability of mixed states for an arbitrary s. These results indicated that this bi-stability for 
mixed states is not dependent on details of the structure in stored patterns. 




V. DISCUSSION 

We have shown that bi-stability of mixed states exists in two typical models, Model 1 and 
Model 2. The reason for this bi-stability, which is based on the SCSNA, can be explained 
as follows. We introduce a model where the contribution of uncondensed patterns to the 
coupling Jij is replaced by the spin glass type interaction, and we call this model Model 3. 
The synaptic coupling of Model 3 is defined as follows, 

'^«. = ^Ee-'%''^ + ^.., (47) 

u=l 

and the symmetric noise 5ij is independently drawn from the following rule, 

S,^-N{0,^^), (48) 
Sij = 5ji, (49) 

where 5 is constant. We obtained the following order parameter equations for this model 



with the SCSNA 13 
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J Dz < ^^'""Y >^i, (50) 
q = j Dz<{Yf>^^, (51) 
U = ^ J Dzz <Y >^i, (52) 



y = F(^e^'"m^'" + ry + az), (53) 

(T=l 

r = 5^f/, (54) 

dz — z'^ 

= yl^exp(^), (55) 

= 6\. (56) 

As you can notice, this model corresponds to the Sherrington-Kirkpatrick (SK) model. In 
contrast to Model 1 or Model 2, a bi-stability of mixed states does not exist in Model 3. 
Note here that the analytical expression of the cross-talk noise variance expressed by eq. 
(^) or eq. ( ^S]) is explicitly expressed as a function of susceptibility U, which is a kind 
of a cross-talk noise-enhancement factor caused by the full feedback nature of the model. 
However, the analytical expression of the cross-talk noise variance for Model 3, a in eq. (pBD, 
is not dependent on the susceptibility U. The above results show that the bi-stability is due 
to not only the hierarchy of stored patterns but also analytical expression of the cross-talk 
noise variance, which is a function of susceptibility for the model. Naturally, the bi-stability 
may exist in more complex hierarchies because the corresponding cross-talk noise variance is 
explicitly expressed by a function of susceptibility as shown in eq. (0). From these results, 
we also found that the bi-stability of mixed states does not exist in a strongly diluted system 

0- 

Finally, we discuss the relation between findings concerning the retrieval process in Fig. 
4 and the physiological experiments of face-responsive neurons in the IT cortex by Sugase 
et al [jl| . Recently, Sugase et al. have analyzed the time-course of information carried by the 
firing of face-responsive neurons in the IT cortex, while performing a fixation task of monkey 
and human faces with various expressions, and simple geometrical shapes. They found that 
the initial transient firing correlated well with a rough categorization (e.g., face vs. non-face 
stimuli). Their results suggest that the neuron firing pattern is initially a superposition of 
patterns representing different faces or expressions, but it then converges to a single pattern 
representing a specific face or expression. We found that the retrieval dynamics of Model 1, 
shown in Fig. ^, can qualitatively replicate the temporal dynamics of face-responsive neurons 
as follows |]TB|]. Initially, the network state approaches a mixed state (77^ or 77^ ) that is a 



superposition of patterns representing different persons or expressions. After that it diverges 
from the mixed state, and finally converges to a single memory pattern (^^'^ ) representing 
a specific person or expression as shown in Fig. H. From the above results, we expect that 



the present system may mimic the temporal dynamics of the face-responsive neurons [|15 
Details will be discussed elsewhere. 
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APPENDIX A: 

The internal potential of the i-the neuron hi in the equilibrium state of eq. (|15D is 

N 

&N &N 

= J2 i^p^i^^ ~irr^ ^p^'- ^^^^ 
p=i p=i 

The output Xi can be formally expressed as 

= F(^Kpafm^- — ^KpXi), (A3) 
p=i p=i 

aN 

= F{Y^K,aPmP), (A4) 
p=i 

where the function F{-) will be determined later as shown in eq. (26). Here, we assume that 
the equilibrium state x has nonzero overlaps with s rotated patterns [p = 1,2, ...,s). 
The residual overlap m'' ~ 0{1/^/N), {s + 1 < p < aN) can be derived by using Taylor 
expansion 



m 



p 



1 N aN 

T7E<^(E V^f'^^'), (A5) 

i=l p'=l 
1 N 

J2afxl'^ + K,UmP, (A6) 
i=i 

1 N 

E^f4'^ (A7) 



Nil - n,U) ^ 



where 



aN 

F{J2K,,afmP'), (A8) 

aN 

F(^ V^f^"'), (A9) 
N 



1 ^ 

u = i^Y.4'^- (Aio) 



i=l 
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Substituting eq. (|A7|) into eq. ([A2|) , we obtain 

s 

hi = Y1 i^p^i^^ + + Zi, (All) 
p=i 

where V is defined as 

l_ ^ t^.^U 
N 



r--i:,T^. (A12) 



and is the effective noise so that, 

-I aN N 



p=s+l j^i P 

Note that is a summation of uncondensed patterns with < z >= and < z"^ >= ar, 

^ &N 2 ^ N 

Let us express the sum of Kp in terms of an integration along continuous eigenvalue = 
Kp for p,N oo, 

JO 1 — K,(u)UY 



1 ^ 



? = ]^E(^0'- (A16) 



i=l 



T = afdu-^^^. (A17) 

JO 1 — K{U)U 

We can obtain eqs. (0) - (|28|) by replacing with y. 

APPENDIX B: 

Let us transform the order parameter equations for the mixed state in Model 1 into the 
equation in one variable. We introduce y'^ in the following equation, 

r = (Bl) 

Since y'^ = y {u = 1, 2, 3, s) holds in the mixed state, we can express the order parameter 
equations for the mixed state by the following equation in y, 

^y,a,b,s) = 0, (B2) 
t^^\{w{y) - KO[y)rJ 

9{y) ^J^< eM-dlv^^n') >.i, (B4) 
V vra ^ s 

w{y) ^ < e^^^^idiven m 
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where is the z/-th eigenvalue of the matrix B. The order parameter equations for the 
mixed states in Model 2 can be also transformed into the equation in one variable as follows, 



^{y,a,b,s) = V2^y{J— < exp(-(^|/e''^)') >£i - < e''^erf(^ t/C'''^) >£i 

= 0. (B6) 
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FIGURES 



FIG. 1. Variation of critical loading rate of memory pattern or mixed states with b. Two kinds 
of mixed states coexist in triangular region. 

FIG. 2. (a) Typical example (6 = 0.61, s = 3) of a dependency of overlap between equilibrium 
state and retrieval pattern, (b) Solutions of transformed order parameter equation in one variable 
around bi-stable region when < aj? 

FIG. 3. (a) Typical example (6 = 0.55, s = 3) of a dependency of overlap between equilibrium 
state and retrieval pattern, (b) Solutions of transformed order parameter equation in one variable 
around bistable region when < 

FIG. 4. Retrieval processes from various initial states in computer simulation when b = 0.475, 
s = 3, a = 0.0087, N = 40000. 
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